Realistic Audio Communication in a Three Dimensional Computer-Generated Virtual Environment

ABSTRACT

A participant in a three dimensional computer-generated virtual environment is able to control a dispersion pattern of his Avatar&#39;s voice such that the Avatar&#39;s voice may be directionally enhanced using simple controls. The audio dispersion envelope is designed to extend further in front of the Avatar and less to the sides and rear of the Avatar. The audio dispersion envelope may be static or controllable by the participant to enable the distance that the Avatar&#39;s voice travels within the virtual environment to be adjusted. This enables the Avatar to whisper or “shout” in the virtual environment such that other Avatars normally outside of hearing range of the Avatar may selectively receive audio generated by the user. Separate audio streams are mixed for each user from audio generated by users with Avatars within their Avatar&#39;s dispersion envelope. The volume of audio from a user in the mixed audio stream depends on the separation of the Avatars within the virtual environment.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 61/037,447, filed Mar. 18, 2008, entitled “Method and Apparatus ForProviding 3 Dimensional Audio on a Conference Bridge”, the content ofwhich is hereby incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to virtual environments and, moreparticularly, to a method and apparatus for implementing realistic audiocommunications in a three dimensional computer-generated virtualenvironment.

2. Description of the Related Art

Virtual environments simulate actual or fantasy 3-D environments andallow for many participants to interact with each other and withconstructs in the environment via remotely-located clients. One contextin which a virtual environment may be used is in connection with gaming,although other uses for virtual environments are also being developed.

In a virtual environment, an actual or fantasy universe is simulatedwithin a computer processor/memory. Multiple people may participate inthe virtual environment through a computer network, such as a local areanetwork or a wide area network such as the Internet. Each player selectsan “Avatar” which is often a three-dimensional representation of aperson or other object to represent them in the virtual environment.Participants send commands to a virtual environment server that controlsthe virtual environment to cause their Avatars to move within thevirtual environment. In this way, the participants are able to causetheir Avatars to interact with other Avatars and other objects in thevirtual environment.

A virtual environment often takes the form of a virtual-reality threedimensional map, and may include rooms, outdoor areas, and otherrepresentations of environments commonly experienced in the physicalworld. The virtual environment may also include multiple objects,people, animals, robots, Avatars, robot Avatars, spatial elements, andobjects/environments that allow Avatars to participate in activities.Participants establish a presence in the virtual environment via avirtual environment client on their computer, through which they cancreate an Avatar and then cause the Avatar to “live” within the virtualenvironment.

As the Avatar moves within the virtual environment, the view experiencedby the Avatar changes according to where the Avatar is located withinthe virtual environment. The views may be displayed to the participantso that the participant controlling the Avatar may see what the Avataris seeing. Additionally, many virtual environments enable theparticipant to toggle to a different point of view, such as from avantage point outside of the Avatar, to see where the Avatar is in thevirtual environment.

The participant may control the Avatar using conventional input devices,such as a computer mouse and keyboard. The inputs are sent to thevirtual environment client, which forwards the commands to one or morevirtual environment servers that are controlling the virtual environmentand providing a representation of the virtual environment to theparticipant via a display associated with the participant's computer.

Depending on how the virtual environment is set up, an Avatar may beable to observe the environment and optionally also interact with otherAvatars, modeled objects within the virtual environment, robotic objectswithin the virtual environment, or the environment itself (i.e. anAvatar may be allowed to go for a swim in a lake or river in the virtualenvironment). In these cases, client control input may be permitted tocause changes in the modeled objects, such as moving other objects,opening doors, and so forth, which optionally may then be experienced byother Avatars within the virtual environment.

“Interaction” by an Avatar with another modeled object in a virtualenvironment means that the virtual environment server simulates aninteraction in the modeled environment, in response to receiving clientcontrol input for the Avatar. Interactions by one Avatar with any otherAvatar, object, the environment or automated or robotic Avatars may, insome cases, result in outcomes that may affect or otherwise be observedor experienced by other Avatars, objects, the environment, and automatedor robotic Avatars within the virtual environment.

A virtual environment may be created for the user, but more commonly thevirtual environment may be persistent, in which it continues to existand be supported by the virtual environment server even when the user isnot interacting with the virtual environment. Thus, where there is morethan one user of a virtual environment, the environment may continue toevolve when a user is not logged in, such that the next time the userenters the virtual environment it may be changed from what it lookedlike the previous time.

Virtual environments are commonly used in on-line gaming, such as forexample in online role playing games where users assume the role of acharacter and take control over most of that character's actions. Inaddition to games, virtual environments are also being used to simulatereal life environments to provide an interface for users that willenable on-line education, training, shopping, and other types ofinteractions between groups of users and between businesses and users.

As Avatars encounter other Avatars within the virtual environment, theparticipants represented by the Avatars may elect to communicate witheach other. For example, the participants may communicate with eachother by typing messages to each other or audio may be transmittedbetween the users to enable the participants to talk with each other.

Although great advances have happened in connection with visualrendering of Avatars and animation, the audio implementation has laggedand often the audio characteristics of a virtual environment are notvery realistic. Accordingly, it would be advantageous to be able toprovide a method and apparatus for implementing more realistic audiocommunications in a three dimensional computer-generated virtualenvironment.

SUMMARY OF THE INVENTION

A method and apparatus for implementing realistic audio communicationsin a three dimensional computer-generated virtual environment isprovided. In one embodiment, a participant in a three dimensionalcomputer-generated virtual environment is able to control a dispersionpattern of his Avatar's voice such that the Avatar's voice may bedirectionally enhanced using simple controls. In one embodiment, anaudio dispersion envelope is designed to extend further in a directionin front of the Avatar and in a smaller direction to the sides and rearof the Avatar. The shape of the audio dispersion envelope may beaffected by other aspects of the virtual environment such as ceilings,floors, walls and other logical barriers. The audio dispersion envelopemay be static or optionally controllable by the participant to enablethe Avatar's voice to be extended outward in front of the Avatar. Thisenables the Avatar to “shout” in the virtual environment such that otherAvatars normally outside of hearing range of the Avatar the User canstill hear the user. Similarly, the volume level of the audio may bereduced to allow the Avatars to whisper or adjusted based on therelative position of the Avatars and directions in which the Avatars arefacing. Individual audio streams may be mixed for each user of thevirtual environment depending on the position and orientation of theuser's Avatar in the virtual environment, the shape of the user'sdispersion envelope, and which other Avatars are located within theuser's user dispersion envelope.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the present invention are pointed out with particularity inthe appended claims. The present invention is illustrated by way ofexample in the following drawings in which like references indicatesimilar elements. The following drawings disclose various embodiments ofthe present invention for purposes of illustration only and are notintended to limit the scope of the invention. For purposes of clarity,not every component may be labeled in every figure. In the figures:

FIG. 1 is a functional block diagram of a portion of an example systemenabling users to have access to three dimensional computer-generatedvirtual environment;

FIG. 2 is a two dimensional view of users in an example threedimensional computer-generated virtual environment and showing a normalaudio dispersion envelope of the users in the three dimensional computergenerated virtual environment;

FIG. 3 is a two dimensional view of users in an example threedimensional computer-generated virtual environment and showing adirectional audio dispersion envelope of the users in the threedimensional computer generated virtual environment;

FIGS. 4-5 show two examples of user controllable audio dispersionenvelopes to enable the user to project his voice in the threedimensional computer-generated virtual environment according to anembodiment of the invention;

FIGS. 6 and 7 show interaction of the audio dispersion envelope withobstacles in an example three dimensional computer-generated virtualenvironment according to an embodiment of the invention;

FIG. 8 is a flow chart showing a process of implementing realistic audiocommunications in a three dimensional computer-generated virtualenvironment;

FIG. 9 is a functional block diagram showing components of the system ofFIG. 1 interacting to enable audio to be transmitted between users ofthe three dimensional computer-generated virtual environment accordingto an embodiment of the invention;

FIG. 10 is a diagram of three dimensional coordinate space showingdispersion envelopes in three dimensional space; and

FIG. 11 is a three dimensional view of a virtual environment.

DETAILED DESCRIPTION

The following detailed description sets forth numerous specific detailsto provide a thorough understanding of the invention. However, thoseskilled in the art will appreciate that the invention may be practicedwithout these specific details. In other instances, well-known methods,procedures, components, protocols, algorithms, and circuits have notbeen described in detail so as not to obscure the invention.

FIG. 1 shows a portion of an example system 10 showing the interactionbetween a plurality of users 12 and one or more virtual environments 14.A user may access the virtual environment 14 from their computer 22 overa packet network 16 or other common communication infrastructure. Thevirtual environment 14 is implemented by one or more virtual environmentservers 18. Audio may be transmitted between the users 12 by one or morecommunication servers 20.

The virtual environment may be implemented as using one or moreinstances, each of which may be hosted by one or more virtualenvironment servers. Where there are multiple instances, the Avatars inone instance are generally unaware of Avatars in the other instance,however a user may have a presence in multiple worlds simultaneouslythrough several virtual environment clients. Conventionally, eachinstance of the virtual environment may be referred to as a separateWorld. In the following description, it will be assumed that the Avatarsare instantiated in the same world and hence can see and communicatewith each other. A world may be implemented by one virtual environmentserver 18, or may be implemented by multiple virtual environmentservers. The virtual environment is designed as a visual representationof a real-world environment that enables humans to interact with eachother and communicate with each other in near-real time.

Generally, a virtual environment will have its own distinct threedimensional coordinate space. Avatars representing users may move withinthe three dimensional coordinate space and interact with objects andother Avatars within the three dimensional coordinate space. The virtualenvironment servers maintain the virtual environment and generate avisual presentation for each user based on the location of the user'sAvatar within the virtual environment. The view may also depend on thedirection in which the Avatar is facing and the selected viewing option,such as whether the user has opted to have the view appear as if theuser was looking through the eyes of the Avatar, or whether the user hasopted to pan back from the Avatar to see a three dimensional view ofwhere the Avatar is located and what the Avatar is doing in the threedimensional computer-generated virtual environment.

Each user 12 has a computer 22 that may be used to access thethree-dimensional computer-generated virtual environment. The computer22 will run a virtual environment client 24 and a user interface 26 tothe virtual environment. The user interface 26 may be part of thevirtual environment client 24 or implemented as a separate process. Aseparate virtual environment client may be required for each virtualenvironment that the user would like to access, although a particularvirtual environment client may be designed to interface with multiplevirtual environment servers. A communication client 28 is provided toenable the user to communicate with other users who are alsoparticipating in the three dimensional computer-generated virtualenvironment. The communication client may be part of the virtualenvironment client 24, the user interface 26, or may be a separateprocess running on the computer 22.

The user may see a representation of a portion of the three dimensionalcomputer-generated virtual environment on a display/audio 30 and inputcommands via a user input device 32 such as a mouse, touch pad, orkeyboard. The display/audio 30 may be used by the user totransmit/receive audio information while engaged in the virtualenvironment. For example, the display/audio 30 may be a display screenhaving a speaker and a microphone. The user interface generates theoutput shown on the display under the control of the virtual environmentclient, and receives the input from the user and passes the user inputto the virtual environment client. The virtual environment clientenables the user's Avatar 34 or other object under the control of theuser to execute the desired action in the virtual environment. In thisway the user may control a portion of the virtual environment, such asthe person's Avatar or other objects in contact with the Avatar, tochange the virtual environment for the other users of the virtualenvironment.

Typically, an Avatar is a three dimensional rendering of a person orother creature that represents the user in the virtual environment. Theuser selects the way that their Avatar looks when creating a profile forthe virtual environment and then can control the movement of the Avatarin the virtual environment such as by causing the Avatar to walk, run,wave, talk, or make other similar movements. Thus, the block 34representing the Avatar in the virtual environment 14, is not intendedto show how an Avatar would be expected to appear in a virtualenvironment. Rather, the actual appearance of the Avatar is immaterialsince the actual appearance of each user's Avatar may be expected to besomewhat different and customized according to the preferences of thatuser. Since the actual appearance of the Avatars in the threedimensional computer-generated virtual environment is not important tothe concepts discussed herein, Avatars have generally been representedherein using simple geometric shapes such as cubes and diamonds, ratherthan complex three dimensional shapes such as people and animals.

FIG. 2 shows a portion of an example three dimensionalcomputer-generated virtual environment and showing normal audiodispersion envelopes associated with Avatars in the three dimensionalcomputer generated virtual environment. FIG. 2 has been shown in twodimensions for ease of illustration and to more clearly show how audiodispersion occurs. Audio dispersion occurs in the vertical direction aswell in the same manner. To simplify the explanation, most of thedescription uses two-dimensional figures to explain how audio dispersionmay be affected in the virtual environment. It should be remembered thatmany virtual environments are three dimensional and, hence, the audiodispersion envelopes will extend in all three dimensions. Extension ofthe two dimensional audio dispersion envelopes to three dimensions isstraightforward, and several examples of this are provided in connectionwith FIGS. 10 and 11. Thus, although the description may focus on twodimensions the invention is not limited in this manner as the sameprinciples may be used to control the vertical dispersion (Z coordinatedirection) as may be used to control the X and Y coordinate dispersionsproperties.

For example, FIG. 10 shows another example in which a first Avatar islocated at XYZ coordinates 2, 2, 25, and a second Avatar is located atXYZ coordinates 25, 25, 4. Viewing only the X and Y coordinates, the twoAvatars are approximately 18.34 units apart. If the audio dispersionenvelope is 20 units, then looking only at the X and Y coordinates thetwo Avatars should be able to talk with each other. However, when the Zcoordinate is factored in, the separation between the two Avatars in thethree dimensional space is 28.46 which is well beyond the 20 unitdispersion envelope reach. Accordingly, in a three dimensional virtualenvironment it may often be necessary to consider the Z coordinateseparation of the two Avatars as well as the X and Y coordinateseparation when determining whether the two Avatars can talk with eachother. A mixed audio stream will be created for each user based on theposition of the user's Avatar and the dispersion envelope for theAvatar, so that audio from each of the users represented by an Avatarwithin the user's dispersion envelope can be mixed and provided to theuser. In this way individual audio streams may be created for each userso that the user can hear audio from other users proximate their Avatarin the virtual environment. The volume of a particular user will beadjusted during the mixing process so that audio from close Avatars islouder than audio from Avatars that are farther away.

As shown in FIG. 2, in this example it has been assumed that fiveAvatars 34 are present in the viewable area of virtual environment 14.The Avatars have been labeled A through E for purposes of discussion.Typically each Avatar would be controlled by a separate user, althoughthere may be instances where a user could control more than one Avatar.In the figure, the arrow shows the direction that the avatar is facingin the virtual environment.

In the example shown in FIG. 2, users associated with Avatars aregenerally allowed to talk with each other as long as they are within aparticular distance of each other. For example, Avatar A is not withinrange of any other Avatar and, accordingly, the user associated withAvatar A is not able to talk to the users associated with any of theother Avatars. Similarly, Avatars D and E are not sufficiently close totalk with each other, even though they are looking at each other.Avatars B and C can hear each other, however, since they aresufficiently close to each other. In the example shown in FIG. 2, theaudio dispersion envelopes for each of the Avatars are spherical(circular in two dimensional space) such that an Avatar can communicatewith any Avatar that is within a particular radial distance. When theuser associated with the Avatar speaks, the other Avatars within theradial distance will hear the user. If the Avatars are too far apartthey are not able to “talk” within the virtual environment and, hence,audio packets/data will not be transmitted between the users.

If users are closer together, the users will be able to hear each othermore clearly, and as the users get farther apart the volume of the audiotapers off until, at the edge of the dispersion envelope, thecontribution of a user's audio is reduced to zero. In one embodiment,the manner in which the volume of a user's contribution is determined ona linear basis. Thus, looking at the example shown in FIG. 3, the user Ais located within a dispersion envelope. The volume close to the user ishighest, as you get farther away from the user A within the dispersionenvelope the volume of contribution tapers off on a linear basis untilthe edge of the dispersion envelope is reached.

Audio is mixed individually for each user of the virtual environment.The particular mix of audio will depend on which other users are withinthe dispersion envelope for the user, and their location within thedispersion envelope. The location within the dispersion envelope affectsthe volume with which that user's audio will be presented to the userassociated with the dispersion envelope. Since the audio is mixedindividually for each user, an audio bridge is not required per user,but rather an audio stream may be created individually for each userbased on which other users are proximate that user in the virtualenvironment.

FIG. 3 shows an embodiment of the invention in which the audiodispersion envelope may be shaped to extend further in front of theAvatar and to a lesser distance behind and to the left and right sidesof the Avatar. This enables the Avatar to talk in a particular directionwithin the virtual environment while minimizing the amount of noise theAvatar generates for other users of the virtual environment. Thus, theability of a user to talk with other users is dependent, in part, on theorientation of the user's Avatar within the virtual environment. Thismirrors real life, where it is easier to hear someone who is facing youthan to hear the same person talking at the same volume, but facing adifferent direction. As a result, the audio dispersion envelope may becontrolled by the user using simple controls, such as the controls usedto control the direction in which the Avatar is facing. Optionally, asdiscussed below, other controls may also be used to control the volumeor distance of the audio dispersion envelope to further enhance theuser's audio control in the virtual environment.

In the example shown in FIG. 3, the Avatars are in the same position asthey are in the example shown in FIG. 2. However, the audio dispersionenvelopes have been adjusted to extend further in the direction that theAvatar is facing and to extend to a lesser extent to the sides of theAvatar and to the rear of the Avatar. This affects which Avatars areable to communicate with each other. For example, as shown in FIG. 3,Avatars B and C are not able to talk with each other, even though theyare standing relatively close to each other, since they are not facingeach other. Similarly, Avatars D and E are able to talk with each othersince they are facing each other, even though they are somewhat fartherapart than Avatars B and C. Avatar A is still alone and, hence, cannottalk to any other Avatars.

In the embodiment shown in FIG. 3, the Avatars must be within eachother's audio dispersion envelope for audio to be transmitted betweeneach other. An alternative may be to enable audio to be transmittedwhere at least one of the Avatars is within the dispersion envelope ofthe other Avatar. For example, if Avatar B is within the dispersionenvelope of Avatar C, but Avatar C is not in the dispersion envelope ofAvatar B, audio may be transmitted between the users associated withAvatars C and B. Optionally, audio may be transmitted in one directionsuch that the user associated with Avatar C can hear the user associatedwith Avatar B, but the user associated with Avatar B cannot hear theuser associated with Avatar C.

The shape of the audio dispersion envelope may depend on the preferencesof the user as well as the preferences of the virtual environmentprovider. For example, the virtual environment provider may provide theuser with an option to select an audio dispersion shape to be associatedwith their Avatar when they enter the virtual environment. The audiodispersion shape may be persistent until adjusted by the user. Forexample, the user may select a voice for their Avatar such that someusers will have robust loud voices while others will have more meek andquiet voices. Alternatively, the virtual environment provider mayprovide particular audio dispersion profiles for different types ofAvatars, for example police Avatars may be able to be heard at a greaterdistance than other types of Avatars. Additionally, the shape of theaudio dispersion envelope may depend on other environmental factors,such as the location of the Avatar within the virtual environment andthe presence or absence of ambient noise in the virtual environment.

FIG. 4 shows an embodiment in which the Avatar is provided with anoption to increase the volume of their voice so that their voice mayreach farther within the virtual environment. This allows the Avatar to“shout” within the virtual environment. As shown in FIG. 4, assume thatthe user associated with Avatar A would like to talk to the userassociated with Avatar B. User A may face Avatar B in the virtualenvironment by causing his Avatar A to face Avatar B. As discussed abovein connection with FIG. 3, where the audio dispersion profile isdesigned to extend further in front of the Avatar, this action willcause the main audio dispersion profile lobe to extend toward Avatar B.However, as shown in FIG. 4, the main audio dispersion lobe of the audiodispersion profile may be insufficiently large to encompass Avatar B ifAvatars A and B are too far apart in the virtual environment.

According to an embodiment of the invention, user A may “Shout” towardAvatar B to cause the audio dispersion profile to extend further in thedirection of B. The user's intention to shout at B may be indicated bythe user through the manipulation of simple controls. For example, on awheeled mouse, the mouse wheel may be a shout control that the user usesto extend the audio dispersion profile of their Avatar. In thisembodiment, the user may simply scroll the mouse wheel away bycontacting the mouse wheel and pushing forward with their finger. Thisis commonly used to scroll up using the mouse wheel in most commoncomputer user interfaces. Conversely, if the user no longer wants toshout, the user may pull back on the top of the mouse wheel in a motionsimilar to scrolling down with the mouse wheel on most common userinterfaces.

There are times where the user may want to communicate with every userwithin a given volume of the virtual environment. For example a personmay wish to make a presentation to a room full of Avatars. To do this,the user may cause their audio dispersion envelope to increase in alldirections to expand to fill the entire volume. This is shown in FIGS. 4and 5 as dashed line 5. The volume of the user's voice in thisembodiment won't increase, but rather all users within the volume willbe able to hear the user so that they can hear audio generated by theuser. This feature will be referred to herein as “OmniVoice”. When theuser invokes OmniVoice, audio from the user is mixed into the audiostream presented to each of the other users within the volume so thatthe other users can hear the user invoking OmniVoice. When the userinvokes OmniVoice, the appearance of the user may be altered somewhat sothat other people engaged in the virtual environment are aware of whohas invoked the OmniVoice feature.

A user may use explicit controls to invoke OmniVoice or, preferably,OmniVoice may be invoked intrinsically based on the location of theAvatar within the virtual environment. For example, the user may walk upto a podium on a stage and, the user's presence on the stage, may causeaudio provided by the user to be included in the mixed audio stream ofevery other Avatar within a particular volume of the virtualenvironment.

The mouse wheel may have multiple uses in the virtual environment,depending on the particular virtual environment and the other activitiesavailable to the Avatar. If this is the case, then a combination ofinputs may be used to control the audio dispersion profile. For example,left clicking on the mouse combined with scrolling of the mouse wheel,or depressing a key on the keyboard along with scrolling of the mousewheel may be used to signal that the mouse scrolling action isassociated with the Avatar's voice rather than another action.Alternatively, the mouse wheel may not be used and a key stroke orcombination of keystrokes on the keyboard may be used to extend theaudio dispersion envelope and cause the audio dispersion envelope toreturn to normal. Thus, in addition to implementing directional audiodispersion envelopes based on the orientation of the Avatar within thethree dimensional virtual environment, the user may also warp the audiodispersion envelope in real time to control the Avatar's audiodispersion envelope in the virtual environment.

When the user elects to shout toward another user in the virtualenvironment, the facial expression of the Avatar may change or othervisual indication may be provided as to who is shouting. This enablesthe user to know that they are shouting as well as to enable other usersof the virtual environment to understand why the physics have changed.For example, the Avatar may cup their hands around their mouth toprovide a visual clue that they are shouting in a particular direction.A larger extension of the Avatar's voice may be displayed as an Avataror even by providing a ghost of the Avatar move closer to the new centerof the voice range so that other users can determine who is yelling.Other visual indications may be provided as well.

The user controlled audio dispersion envelope warping may toggle, asshown in FIG. 4, such that the user is either talking normally orshouting depending on the state of the toggle control. Alternatively,the user controlled audio dispersion profile warping may be more gradualand have multiple discrete levels as shown in FIG. 5. Specifically, asshown in FIG. 5, the user may increase the directionality and range ofthe projection of their voice in the three-dimensional computergenerated virtual environment so that the reach of their voice mayextend different distances depending on the extent to which the userwould like to shout in the virtual environment. In FIG. 5, the user hasbeen provided with four discrete levels, and then a fifth level for usein connection with OmniVoice. Other numbers of levels may be used aswell. Additionally, the discretization may be blurred such that thecontrol is closer to continuous depending on the implementation.

In another embodiment, the selection of a person to shout to may beimplemented when the user mouses over another Avatar and depresses abutton, such as left clicking or right clicking on the other Avatar. Ifthe person is within normal talking distance of the other Avatar, audiofrom that other Avatar will be mixed into the audio stream presented tothe user. Audio from other Avatars within listening distance willsimilarly be included in mixed audio stream presented to the user. Ifthe other Avatar is not within listening distance, i.e. is not withinthe normal audio dispersion envelope, the user may be provided with anoption to shout to the other Avatar. In this embodiment the user wouldbe provided with an instruction to double click on the other Avatar toshout to them. Many different ways of implementing the ability to shoutmay be possible depending on the preferences of the user interfacedesigner.

In one embodiment, the proximity of the Avatars may adjust the volume oftheir audio when their audio is mixed into the audio stream for the userso that the user is presented with an audio stream that more closelyresemble normal realistic audio. Other environmental factors maysimilarly affect the communication between Avatars in the virtualenvironment.

Although the previous description has focused on enabling the Avatar toincrease the size of the dispersion envelope to shout in the virtualenvironment, the same controls may optionally also be used to reduce thesize of the dispersion envelope so that the user can whisper in thevirtual environment. In this embodiment, the user may control theirvoice in the opposite direction to reduce the size of the dispersionenvelope so that users must be closer to the user's Avatar tocommunicate with the user. The mouse controls or other controls may beused in this manner to reduce the size of the audio dispersion envelopeas desired.

FIG. 6 and 7 show an embodiment in which the profile of the audiodispersion envelope is affected by obstacles in the virtual environment.Commonly, two Avatars in a virtual environment will be able to hear eachother if they are within talking distance of each other regardless ofthe other objects in the virtual environment. For example, assume thatAvatar A is in one room and that Avatar B is in another room as shown inFIG. 7. Normally, the virtual environment server would enable theAvatars to communicate with each other (as shown by the dashed linerepresenting the dispersion envelope) since they are proximate eachother in the virtual environment. FIG. 11 shows a similar situationwhere two Avatars are closed to each other, but are on different floorsof the virtual environment. From the Avatar's perspective, the twoAvatars are unlikely to be able to see through the wall or floor/ceilingand, hence, enabling the Avatars to hear each other through an obstacleof this nature is unnatural. This feature also enables Avatars to listenin on conversations that are occurring within the virtual environmentbetween other Avatars, without being seen by the other Avatars.

According to an embodiment of the invention, the audio dispersionprofile may be adjusted to account for obstacles in the virtualenvironment. Obstacles may be thought of as creating shadows on theprofile to reduce the distance of the profile in a particular direction.This prevents communication between Avatars where they would otherwisebe able to communicate if not for the imposition of the obstacle. InFIG. 6, for example, Avatar A is in a room and talking through adoorway. The jambs and walls on either side of the door serve asobstacles that partially obstruct the audio dispersion profile of theuser. If the door is closed, this too may affect the dispersionenvelope. Thus, Avatar B who is standing next to the wall will be unableto hear or talk to Avatar A since Avatar B is in the shadow of the walland outside of Avatar A's audio dispersion profile. Avatar C, bycontrast, is in direct line of site of Avatar A through the door and,accordingly, may communicate with Avatar A. Of course, Avatar B maycommunicate with Avatar C and, hence may still hear Avatar B's side ofthe conversation between Avatars A and C. Thus, a particular Avatar maybe only partially included in communications between other parties. Toimplement an embodiment of this nature, a discrete mixer is implementedper client, where essentially all these described attributes andproperties are calculated on a per client basis. The mixer determineswhich audio from adjacent Avatars is able to be heard by the particularuser and mixes the available audio streams accordingly. Implementingthis on a per-client basis is advantageous because no two users are everin exactly the same position at a particular point in time, and hencethe final output of each of the clients will necessarily be different.

FIG. 11 shows two Avatars that are separated vertically as they are onseparate floors. The floor may function in the same manner as the wallsdid in FIGS. 5 and 6. Specifically, if the rooms on the separate floorsare defined as separate volumes, sound from one floor will not enter theother floor so that, even though the two Avatars are very close to eachother in both the X and Y coordinate space, they are separatedvertically such that the two Avatars cannot talk with each other.

The shadow objects may be wholly opaque to transmission of sound or maysimply be attenuating objects. For example, a concrete wall mayattenuate sound 100%, a normal wall may attenuate 90% of sound whileallowing some sound to pass through, and a curtain may attenuate soundonly modestly such as 10%. The level of attenuation may be specifiedwhen the object is placed in the virtual environment.

Audio be implemented using a communication server that is configured tomix audio individually for each user of the virtual environment. Thecommunication server will receive audio from all the users of thevirtual environment and create an audio stream for a particular user bydetermining which of the other users have an Avatar within the user'sdispersion envelope. To enable participants to be selected, a notion ofdirectionality needs to be included in the selection process such thatthe selection process does not simply look at the relative distance ofthe participants, but also looks to see what direction the participantsare facing within the virtual environment. This may be done byassociating a vector with each participant and determining whether thevector extends sufficiently close to the other Avatar to warrantinclusion of the audio from that user in the audio stream. Additionally,if shadows are to be included in the determination, the process may lookto determine whether the vector transverses any shadow objects in thevirtual environment. If so, the extent of the shadow may be calculatedto determine whether the audio should be included. Other ways ofimplementing the audio connection determination process may be used aswell and the invention is not limited to this particular exampleimplementation.

In another embodiment, audio is to be transmitted between two Avatarsmay be determined by integrating attenuation along a vector between theAvatars. In this embodiment, the normal “air” or empty space in thevirtual environment may be provided with an attenuation factor such as5% per unit distance. Other objects within the environment may beprovided with other attenuation factors depending on their intendedmaterial. Transmission of audio between Avatars, in this embodiment, maydepend on the distance between the Avatars, and hence the amount of airthe sound must pass through, and the objects the vector passes through.The strength of the vector, and hence the attenuation able to beaccommodated while still enabling communication, may depend on thedirection the Avatar is facing. Additionally, the user may temporarilyincrease the strength of the vector by causing the Avatar to shout.

In the preceding description, it was assumed that the user could electto shout within the virtual environment. Optionally, that privilege maybe reserved for Avatars possessing particular items within the virtualenvironment. For example, the Avatar may need to find or purchase aparticular item such as a virtual bull horn to enable the Avatar toshout. Other embodiments are possible as well.

FIG. 8 shows a flow chart of a process, portions of which may be used byone or more entities, to determine whether audio should be transmittedbetween particular participants in a three dimensionalcomputer-generated virtual environment. In the process shown in FIG. 8,it will be assumed that the virtual environment server(s) have renderedthe virtual environment (100) so that it is available for use byparticipants. Thus, the virtual environment servers will enable user Ato enter the virtual environment and will render Avatar A for user Awithin the virtual environment (102). Similarly, the virtual environmentservers will enable user B to enter the virtual environment and willrender Avatar B for user B within the virtual environment (102′). Otherusers may have Avatars within the virtual environment as well.

The virtual environment servers will also define an audio dispersionenvelope for Avatar A which specifies how the Avatar will be able tocommunicate within the virtual environment (104). Each Avatar may have aset pre-defined audio dispersion envelope which is a characteristic ofall Avatars within the virtual environment, or the virtual environmentservers may define custom specific audio dispersion envelopes for eachuser. Thus, the step of defining audio dispersion envelopes may besatisfied by specifying that the Avatar is able to communicate withother Avatars that are located a greater distance in front of the Avatarthan other Avatars located in other directions relative to the Avatar.

Automatically, or upon initiation of the user controlling Avatar A, thevirtual environment server will determine whether Avatar B is withinaudio dispersion envelope for Avatar A (106). This may be implemented,for example, by looking to see whether Avatar A is facing Avatar B, andthen determining how far distance Avatar B is from Avatar A in thevirtual environment. If Avatar B is within the audio dispersion envelopeof Avatar A, the virtual environment server will enable audio fromAvatar B to be included in the audio stream transmitted to the userassociated with Avatar A.

If Avatar B is not within the audio dispersion envelope of Avatar A, theuser may be provided with an opportunity to control the shape of theaudio dispersion envelope such as by enabling the user to shout towardAvatar B. In particular, user A may manipulate their user interface tocause Avatar A to shout toward Avatar B (110). If user A properlysignals via their user interface that he would like to shout towardAvatar B, the virtual environment server will enlarge the audiodispersion envelope for the Avatar in the virtual environment in thedirection of the shout (112).

The virtual environment server will then similarly determine whetherAvatar B is within the enlarged audio dispersion envelope for Avatar A(114). If so, the virtual environment server will enable audio to betransmitted between the users associated with the Avatars (116). If not,the two Avatars will need to move closer toward each other in thevirtual environment to enable audio to be transmitted between the usersassociated with Avatars A and B (118).

FIG. 9 shows a system that may be used to implement realistic audiowithin a virtual environment according to an embodiment of theinvention. As shown in FIG. 9, users 12 are provided with access to avirtual environment 14 that is implemented using one or more virtualenvironment servers 18.

Users 12A, 12B are represented by avatars 34A, 34B within the virtualenvironment 14. When the users are proximate each other and facing eachother, an audio position and direction detection subsystem 64 willdetermine that audio should be transmitted between the users associatedwith the Avatars. Audio will be mixed by mixing function 78 to provideindividually determined audio streams to each of the Avatars.

In the embodiment shown in FIG. 9, the mixing function is implemented atthe server. The invention is not limited in this manner as the mixingfunction may instead be implemented at the virtual environment client.In this alternative embodiment, audio from multiple users would betransmitted to the user's virtual environment client and the virtualenvironment client would select particular portions of the audio to bepresented to the user. Implementing the mixing function at the serverreduces the amount of audio that must be transmitted to each user, butincreases the load on the server. Implementing the mixing function atthe client distributes the load for creating individual mixed audiostreams and, hence, is easier on the server. However, it requiresmultiple audio streams to be transmitted to each of the clients. Theparticular solution may be selected based on available bandwidth andprocessing power.

The functions described above may be implemented as one or more sets ofprogram instructions that are stored in a computer readable memory andexecuted on one or more processors within on one or more computers.However, it will be apparent to a skilled artisan that all logicdescribed herein can be embodied using discrete components, integratedcircuitry such as an Application Specific Integrated Circuit (ASIC),programmable logic used in conjunction with a programmable logic devicesuch as a Field Programmable Gate Array (FPGA) or microprocessor, astate machine, or any other device including any combination thereof.Programmable logic can be fixed temporarily or permanently in a tangiblemedium such as a read-only memory chip, a computer memory, a disk, orother storage medium. All such embodiments are intended to fall withinthe scope of the present invention.

It should be understood that various changes and modifications of theembodiments shown in the drawings and described in the specification maybe made within the spirit and scope of the present invention.Accordingly, it is intended that all matter contained in the abovedescription and shown in the accompanying drawings be interpreted in anillustrative and not in a limiting sense. The invention is limited onlyas defined in the following claims and the equivalents thereto.

1. A method of selectively enabling audio to be transmitted betweenusers of a three dimensional computer-generated virtual environment, themethod comprising the steps of: determining a position of a first Avatarin the virtual environment; determining an orientation of the firstAvatar in the virtual environment to determine which way the firstAvatar is facing in the virtual environment; enabling audio from thesecond user associated with a second Avatar to be included in a mixedaudio stream for the first user associated with the first Avatar if thesecond Avatar is located within a first distance of the first Avatar inthe virtual environment and the first Avatar is facing the secondAvatar; and not enabling audio from the second user associated with asecond Avatar to be included in a mixed audio stream for the first userassociated with the first Avatar if the second Avatar is located withinthe first distance of the first Avatar in the virtual environment butthe first Avatar is not facing the second Avatar.
 2. The method of claim1, further comprising the step of defining a audio dispersion envelopefor the first Avatar, such that if the second Avatar is located withinthe audio dispersion envelope then audio from the user associated withthe second Avatar will be included in the mixed audio stream for thefirst user, and if the second Avatar is located outside the audiodispersion envelope, then audio from the user associated with the secondAvatar will not be included in the mixed audio stream for the firstuser.
 3. The method of claim 2, wherein the audio dispersion envelopefor the first Avatar is controllable by the first user associated withthe first Avatar.
 4. The method of claim 3, wherein the audio dispersionenvelope for the first Avatar may be increased to extend a greaterdirection toward a direction that the first Avatar is facing to enablethe first Avatar to shout in the virtual environment.
 5. The method ofclaim 3, wherein the audio dispersion envelope for the first Avatar maybe decreased to extend a lesser direction toward a direction that thefirst Avatar is facing to enable the first Avatar to whisper in thevirtual environment.
 6. The method of claim 3, wherein the audiodispersion envelope for the first Avatar may be increased to extend tocoincide with a volume of the virtual environment, such that any audiogenerated by the first user associated with the first Avatar will beincluded in audio streams mixed for all other users associated withAvatars that are located within the volume of the virtual environment.7. The method of claim 6, wherein the audio dispersion envelope for thefirst Avatar is automatically increased to extend to coincide with thevolume of the virtual environment when the first Avatar enters aparticular location within the virtual environment.
 8. The method ofclaim 3, wherein the audio dispersion envelope is controllable by thefirst user via interaction with a mouse wheel.
 9. The method of claim 1,the method further comprising the step of determining whether there areobstacles between the first Avatar and the second Avatar in the virtualenvironment, and selectively not including audio from the second userassociated with the second Avatar in the mixed audio stream for thefirst user associated with the first Avatar if there are sufficientobstacles between the first and second Avatars.
 10. The method of claim9, wherein the obstacle is a wall.
 11. The method of claim 2, wherein ashape of the audio dispersion envelope for the first Avatar is dependenton Avatar's location within environment.
 12. A computer programincluding data and instructions that are stored on a computer readablememory which, when loaded on a computer processor, enables the computerto implement a method of enabling realistic audio to be simulated in athree dimensional computer-generated virtual environment, the methodcomprising the steps of: defining a first audio dispersion envelopewithin the three dimensional computer-generated virtual environment fora first Avatar associated with a first user; and selecting audio fromother users associated with Avatars located within the audio dispersionenvelope for inclusion in a mixed audio stream to be presented to thefirst user.
 13. The computer program of claim 12, wherein the methodfurther comprises the steps of adjusting a shape of the audio dispersionenvelope based on obstacles in the three dimensional computer-generatedvirtual environment.
 14. The computer program of claim 12, furthercomprising defining an attenuation factor for items in the virtualenvironment to determine a size and shape of the dispersion envelope.15. The computer program of claim 14, further comprising enabling thefirst user to control the audio dispersion envelope.
 16. The computerprogram of claim 15, wherein the step of enabling the user to controlthe audio dispersion envelope enables the user to enlarge the audiodispersion envelope to include audio from Avatars that are farther awayso that the Avatar can shout in the three dimensional computer-generatedvirtual environment.
 17. The computer program of claim 16, wherein thestep of enabling the user to control the audio dispersion envelopeenables the user to reduce the audio dispersion envelope to preventAvatars from being included in the audio dispersion envelope so that theAvatar can whisper in the three dimensional computer-generated virtualenvironment.
 18. The computer program of claim 12, wherein if a secondAvatar is within the dispersion envelope for the first Avatar, then auser associated with the second Avatar can hear audio from the firstuser.
 19. The computer program of claim 18, further comprising the userto invoke a feature that will enable audio from the first user to beincluded in audio streams provided to all users associated with Avatarswithin a volume of the virtual environment.
 20. The computer program ofclaim 19, wherein the feature is intrinsically invoked when the firstAvatar enters a defined area of the virtual environment.